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This study used administrative data from Wisconsin Virtual School to identify patterns 
of student engagement in online courses (defined as the amount of time students were 
logged in to their course each week and how this varied over time). The study also examined 
whether the patterns were associated with course outcomes (defined as the percentage 
of possible points earned and the percentage of course activities completed). Six patterns 
of engagement were identified, and most students in five of the six engagement groups 
earned a high enough percentage of possible points to pass their online course. Students 
with low but steady engagement in their online course had better outcomes than students 
with low initial engagement that diminished throughout their course. Students who engaged 
in their online course for two or more hours per week had better outcomes than students 
who engaged for fewer than two hours per week. 


This brief summarizes findings of Pazzaglia, A. M., Clements, M., Lavigne, H. J., & Stafford, E. T. 
(2016). An analysis of student engagement patterns and online course outcomes in Wisconsin (REL 2016- 
147), Washington, DC: U.S. Department of Education, Institute of Education Sciences, Nation- 
al Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory 
Midwest. That report is available at http://ies.ed.gov/ncee/edlabs/projects/project.asp?projectlD=4514. 
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Why this study? 


Student enrollment in online courses has increased in the past 15 years and continues to grow (Watson, 
Pape, Murin, Gemin, & Vashaw, 2014). However, little is known about students’ educational experiences 
or online course outcomes. Student attrition and course failure are problems for some online learning pro- 
grams (Angelino, Williams, & Natvig, 2007; Carr, 2000; Freidhoff, 2015; Freidhoff, DeBruler, & Kennedy, 
2014; Roblyer, 2006; Roblyer & Davis, 2008; Rovai & Wighting, 2005; Simpson, 2004). A recent study 
found that two of the greatest challenges for online teachers at Wisconsin Virtual School — a state-level 
online learning program for middle and high school students — were keeping students engaged throughout 
the course and getting students to complete the course (Zweig, Stafford, Clements, & Pazzaglia, 2015). 

This study builds on recent research using learning management system and student information system 
data to identify factors that predict student success in online courses. Hung, Hsu, and Rice (2012) found that 
students with higher engagement (as measured by the total number of learning management system activ- 
ities across the span of the course, including frequency of logins, frequency of modules accessed, frequency 
of clicks, and frequency of discussion board posts) tended to have higher final grades. Liu and Cavanaugh 
(2011) found that the number of times a student logged in to the learning management system throughout 
the course and the total number of minutes a student was logged in throughout the course were associated 
with students’ final scores in two online biology courses offered through a Midwestern state virtual school. 
They also found that the total number of minutes that a student was logged in throughout the course was 
the strongest predictor of final scores in a model that accounted for student demographics and the number of 
teacher comments throughout the course. Because these studies were correlational in nature, it is not possL 
ble to determine the direction or extent of causality between engagement and course outcomes. It is possible 
that factors such as motivation or academic ability cause both student engagement and course outcomes. 

A collaboration between the Midwest Virtual Education Research Alliance and Regional Educational Lab- 
oratory Midwest, the current study builds on these earlier studies by identifying distinct patterns of student 
engagement within online courses over time and exploring whether the patterns are associated with online 
course outcomes. It used learning management system and student information system data collected by 
Wisconsin Virtual School, a state-level online learning program that partners with districts in the state to 
offer supplemental online courses for middle and high school students. The goal of the study was to deter- 
mine whether patterns of student engagement (defined as the total amount of time a student was logged 
in to the course each week) are associated with the percentage of possible points earned in the course and 
the percentage of course activities completed (see box 1 for a summary of the data and methodology). Such 
information may help inform policymakers, state education agencies, local education agencies, and online 
course providers as they develop or refine online programs and promote student success in online courses. 
In addition, the study provides a framework for how online learning programs can use the data available in 
their learning management and student information systems to examine student engagement and online 
course outcomes. 


What the study found 


This section discusses the key findings of the study. 

Student enrollments In online courses followed one of six engagement patterns 

The amount of time a student was logged in to the online course each week was best represented by a 
model with six distinct patterns of student engagement (figure 1). The two most common patterns each 
accounted for nearly 40 percent of student enrollments. 
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Box 1. Data and methodology 

The study sample included all Wisconsin Virtual School Advanced Placement, core, and elective high school 
course enrollments from the fall 2014 semester. This sampling strategy excluded enrollments in credit recovery 
courses because students have 12 weeks to complete those courses, compared with five months to com- 
plete other types of courses. The sample Included 1,512 student enrollments In 109 online courses, with 
1,179 unique students, 170 of whom took more than one online course. Fifteen percent of enrollments were 
In Advanced Placement courses, 36 percent were In core courses, and 49 percent were in elective courses. 
Sixty percent of student enrollments were female students, and 40 percent were male students. The majority of 
student enrollments were in grades 9-12; 11 percent were in grade 9, 25 percent were in grade 10, 28 percent 
were in grade 11, and 32 percent were in grade 12. Three percent of student enrollments were in grades 5-8. 

A statistical approach that analyzes patterns of Individual outcomes across time and classifies the patterns 
into unique groups (group-based trajectory modeling; Nagin, 2005) was employed to estimate the number and 
shapes of engagement patterns. Next, an analysis that accounts for the ways in which students are related to 
each other within online courses (hierarchical linear modeling; Raudenbush & Bryk, 2002) was used to test the 
associations between student engagement patterns and student outcomes, while accounting for student and 
course characteristics. Appendix A of the full report explains the analytic approach in more detail. 


Figure 1. Student enrollments in Wisconsin Virtual School online course followed one of six 
engagement patterns, fall 2014 


Hours per week spent logged in to the online course 



1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 

Weeks in online course 


Note: Analysis is based on 1,512 student enrollments. The maximum number of data points for each student enrollment is 21 
weeks, though some students finished their course in fewer than 21 weeks. This approach uses all available data for each week. 
Values in parentheses are the percentage of the sample that followed each engagement pattern; percentages do not sum to 100 
because of rounding. The engagement patterns depicted were stylized using the predicted values for each week. 

Source: Authors’ analysis of Wisconsin Virtual School data (fall 2014 semester). 


The six patterns of student engagement are: 

• Initial 1.5 hours with decrease (8 percent): Engagement of approximately 1.5 hours per week at the 
beginning of the semester that drops off to near 0 hours midway through the semester. 

• Steady 1.5 hours (39 percent): Steady engagement of approximately 1.5 hours per week, with a 
slight increase toward the end of the semester. 
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• Initial 2 hours with spike (4 percent): Engagement of approximately 2 hours per week at the begin- 
ning of the semester that increases steadily after the midpoint of the semester to nearly 12 hours 
per week in the final week. 

• Steady 2.5 hours (38 percent): Steady engagement of approximately 2.5 hours per week across the 
semester. 

• Steady 4+ hours (8 percent): Consistent engagement of approximately 4 hours or more per week 
across the semester. 

• Variable 6+ hours (2 percent): High but variable engagement ranging from approximately 6 hours 
per week at the beginning, middle, and end of the semester and peaking to near 10 hours between 
those periods (during weeks 3 and 17). 

Most students in five of the six engagement groups earned a high enough percentage of possibie points to pass their 
oniine course 

Most students in the engagement group with average engagement of at least 1.5 hours per week (which 
includes all groups except the initial 1.5 hours with decrease group) earned a high enough percentage of 
possible points to pass their online course, though average total time logged in each week and the change 
in that average over time varied across groups (figure 2). 

Students with iow but steady engagement in their oniine course had better outcomes than students with iow initiai 
engagement that diminished throughout the course 

The steady 1.5 hours group had better course outcomes than the initial 1.5 hours with decrease group after 
student and course characteristics were controlled for (table 1). The average percentage of possible points 


Figure 2. Most students in five of the six engagement groups earned a high enough percentage of 
possibie points to pass their course, though their engagement patterns differed, faii 2014 


■ Percentage of possible points earned in the course ■ Percentage of course activities completed 
Initial 1.5 hours with decrease (n = 122) 

Steady 1.5 hours (n = 592) 

Initial 2 hours with spike (n = 68) 

Steady 2.5 hours (n = 572) 

Steady 4+ hours (n = 122) 

Variable 6+ hours (n = 36) 

0 20 40 60 80 100 

Average student outcome 

Note: Data were statistically adjusted for student demographic characteristics and course type taken (Advanced Placement, core, 
or elective). Wisconsin Virtual School provides the percentage of possible points earned in the course to students' home schools, 
which then assign a letter grade based on the local grading scale, so benchmarks for passing grades can vary across schools or 
districts. For Wisconsin Virtual School, students must complete 95-100 percent of the possible course points to obtain a certifi- 
cate of completion. The home district then determines whether 95-100 percent course activity completion with a passing grade, 
which is determined locally, provides an earned credit or advancement. 

Source: Authors’ analysis of Wisconsin Virtual School data (fall 2014 semester). 
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Table 1. Students who engaged for two or more hours per week outperformed students who 
engaged for fewer than two hours per week, and students who engaged 1.5 hours per week 
steadily across the semester outperformed students whose engagement decreased over the 
semester, fall 2014 



Percentage of possible points 
earned in the course 

Percentage of course 
activities completed 

Engagement group 

Mean^ 

Standard deviation 

Mean^ 

standard deviation 

Initial 1.5 hours with decrease 

4 * * 

35 

28*** 

27 

Steady 1.5 hours 

76 

21 

91 

18 

Initial 2 hours with spike 

85*** 

14 

99** 

10 

Steady 2.5 hours 


13 

98*** 

7 

Steady 4+ hours 

88** 

14 

100** 

10 

Variable 6+ hours 

89* 

8 

100* 

0.5 


* is statistically significantly different from the mean for the steady 1.5 hours group at p < .05; ** is statistically significantly 
different from the mean for the steady 1.5 hours group at p < .01; *** is statistically significantly different from the mean for the 
steady 1.5 hours group at p < .001. 

a. Statistically adjusted for student demographic characteristics and course type taken (Advanced Placement, core, or elective). 
Source: Authors' analysis of Wisconsin Virtual School learning management system data (fall 2014 semester). 


earned in the course was 35 percentage points higher for the steady 1.5 hours group than for the initial 
1.5 hours with decrease group, and the percentage of course activities completed was 63 percentage points 
higher for the steady 1.5 hours group than for the initial 1.5 hours with decrease group. 

Students who engaged in their oniine course for two or more hours per week had better outcomes than students who 
engaged for fewer than two hours per week 

After student and course characteristics were controlled for, students who engaged for 2 hours or more per 
week (the initial 2 hours with spike, steady 2.5 hours, steady 4+ hours, and variable 6+ hours groups) had 
better course outcomes than students who engaged consistently over the semester with less time each week 
(the steady 1.5 hours group; see table 1). 

Of particular interest is the comparison between the steady 1.5 hours group and the steady 2.5 hours group, 
which account for a large majority of the sample and had similar engagement patterns, with the exception 
that the steady 2.5 hours group was logged in an average of approximately one additional hour per week. 
The average percentage of possible points earned in the course was 10 percentage points higher for the 
steady 2.5 hours group than for the steady 1.5 hours group, and the average percentage of course activities 
completed was 7 percentage points higher for the steady 2.5 hours group than for the steady 1.5 hours 
group. The differences were statistically significant. 

Implications of the study findings 


This study provides state policymakers, state education agencies, local education agencies, and online 
learning providers with needed information about student engagement in online courses and how patterns 
of engagement are associated with course outcomes. Because the findings are based on correlational data, 
they cannot be used to make causal connections between student engagement and online course outcomes. 

Wisconsin Virtual School directors and directors of other online learning programs may be able to use infon 
mation from this study as they seek ways to support student success in online courses, especially students in the 
initial 1.5 hours with decrease and steady 1.5 hours engagement groups, which had poorer course outcomes. 
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Supports might include targeting students who log in for fewer than two hours per week to see whether they 
are struggling with course content or contacting the student, parents, or the brick-and-mortar school to 
provide notice that the student is at risk of failing the course based on his or her limited participation. 

Other online learning programs across the country may be able to use the findings as a framework for 
investigating the data available in their learning management systems and student information systems. In 
combination with future studies investigating how much these findings can be generalized to students in 
other online learning programs, the results may be able to inform the provision of support to students who 
demonstrate engagement patterns that are associated with poor course outcomes. 

Study limitations 


The study has three main limitations: 

• The findings may not extend to other types of online courses, including credit recovery courses or 
other online learning programs. 

• The amount of time a student is logged in to a learning management system is an imperfect 
measure of student engagement because students are not necessarily actively working while logged 
in, and some courses may require that students engage in academic activities outside the system. 
The learning management system logs students out after five minutes of inactivity, which should 
reduce some of this error. 

• The analysis methods did not fully account for the complex structure of online course data. The 
analyses accounted for the relationships among students only within online course sections, which 
does not capture all possible relationships among students, onsite facilitators, brick-and-mortar 
schools, online courses, and online teachers. 
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The National Center for Education Evaluation and Regional Assistance (NCEE) conducts unbiased 
large-scale evaluations of education programs and practices supported by federal funds; provides 
research-based technical assistance to educators and policymakers; and supports the synthesis and 
the widespread dissemination of the results of research and evaluation throughout the United States. 
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